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EMMA  :  THE  EXPERT  SYSTEM  FOR 
MUNITION  MAINTENANCE 
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Eglin  Air  Force  Base,  Florida  32542-5434 


ABSTRACT 

EMMA  (Expert  Missile  Maintenance  Aid) 
is  the  result  of  research  sponsored  by  the 
Air  Force  Armament  Laboratory,  Air  Force 
Systems  Command  at  Eglin  Air  Force  Base 
(AFB) ,  Florida.  It  is  a  first  attempt  to 
enhance  maintenance  of  a  tactical  munition 
at  the  field  and  depot  level  by  using 
artificial  intelligence  (AI)  techniques. 
The  ultimate  goal  of  EMMA  is  to  help  a 
novice  maintenance  technician  isolate  and 
diagnose  electronic,  electromechanical, 
and  mechanical  equipment  faults  to  the 
boatd/chassis  level  more  quickly  and  con¬ 
sistently  than  the  best  human  expert  using 
the  best  currently  available  automatic 
test  equipment  (ATE).  To  this  end,  EMMA 
augments  existing  ATE  with  an  expert  sys¬ 
tem  that  captures  the  knowledge  of  design 
and  maintenance  experts. 

This  paper  describes  the  EMMA  pro¬ 
gram.  It  addresses  such  issues  as  how  the 
field-level  expert  system  prototypes  were 
evaluated  as  well  as  the  results  of  the 
evaluations.  Additionally,  current  work 
on  the  depot-level  prototypes  is  discussed 
as  well  as  issues  related  to  using  DOD- 
STD-2167  to  document  the  development  of 
expert  systems.  This  paper  will  briefly 
address  several  study  tasks  performed 
during  EMMA.  The  paper  concludes  with  a 
discussion  of  future  plans  for  a  follow-on 
program  and  other  areas  of  concern. 

IIITRODDCTIOH 

Weapon  systems  of  today  are  undoubt¬ 
edly  benefiting  from  technology  advances 
and  justifiably  so.  Munitions  are  becom¬ 
ing  more  sophisticated  and  "smarter"  as  a 
result  of  this  technology.  Electronically 
sophisticated  munitions  ate  quickly  infil¬ 
trating  the  Department  of  Defense  arsenal 
of  weapons.  Simple  bombs  are  becoming 
relics  of  the  past. 

However,  some  shortcomings  can  be 
associated  with  incorporating  new  technol¬ 
ogy  into  current  and  future  weapon  sys¬ 


tems.  Munition  maintenance  will  surely 
suffer  as  a  consequence  of  this  technol¬ 
ogy.  The  technology  advances  that  have 
improved  the  effectiveness  of  munitions 
are  simultaneously  complicating  the  main¬ 
tenance  of  these  munitions  by  increasing 
the  functionality  of  the  munition  typical¬ 
ly  making  the  munition  harder  to  maintain. 
Munition  test  equipment  and  associated 
software  do  not  adequately  diagnose 
faults.  Automatic  test  equipment  (ATE)  is 
plagued  by  high  false  alarm  rates.  Guid¬ 
ance  and  control  sections  returned  to  the 
depot  ate  currently  experiencing  approxi¬ 
mately  28  to  441  retest  OKs.  Many  faults, 
as  many  as  one  out  of  every  four,  cannot 
be  detected  by  ATE.  Sequential  testing  is 
typically  how  ATE  performs  diagnostic 
testing.  This  limits  the  diagnostic  capa¬ 
bilities  of  the  test  equipment.  Addition¬ 
ally,  ATE  cannot  diagnose  beyond  multiple 
linked  components. 

Another  aspect  of  munition  mainte¬ 
nance  that  must  be  addressed  when  dealing 
with  munition  reliability  is  the  personnel 
shortage.  The  current  shortage  of  skilled 
munition  maintenance  technicians  is  a 
serious  problem.  Demographic  projections 
indicate  that  this  dilemma  will  not  sub¬ 
side  in  the  neat  future.  Since  experi¬ 
enced  technicians  are  able  to  diagnose  a 
fault  quicker  and  more  reliably  than  a 
novice,  the  knowledge  acquire  by  the  expe¬ 
rienced  technician  (expert)  throughout  the 
years  should  be  captured  so  that  this 
knowledge  can  be  used  by  novice  techni¬ 
cians  during  future  diagnostic  sessions. 

Artificial  intelligence  (Al)  technol¬ 
ogy  is  one  approach  to  increasing  the 
reliability  and  maintainability  of  ex¬ 
isting  and  future  weapon  systems.  One 
popular  and  heavily  cited  definition  of 
artificial  intelligence  is  provided  by  Dr. 
Elaine  Rich,  University  of  Texas  at 
Austin.  She  defines  AI  as  follows:  "Arti¬ 
ficial  intelligence  is  the  study  of  how 
computers  do  things  at  which,  at  the  mo¬ 
ment,  people  are  better"  (Rich,  1983:1). 
A  subset  of  AI  is  a  field  called  expert 
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systems.  This  area  of  AI  has  emerged 
recently  with  the  greatest  amount  of  suc- 
ess  (Hayes-Roth  et  al.,  1983:xi).  Donald 
aterman  defines  expert  systems  as  "so¬ 
phisticated  computer  programs  that  manipu¬ 
late  knowledge  to  solve  problems  effi¬ 
ciently  and  effectively  in  a  narrow  prob¬ 
lem  area"  (Waterman,  1986:xvii). 

The  Air  Force  has  recognized  the 
importance  of  increasing  munition  reli¬ 
ability.  In  a  joint  memorandum.  General 
Gabriel,  former  Air  Force  Chief  of  Staff, 
and  Verne  Orr,  former  Secretary  of  the  Air 
Force  stated,  "For  too  long,  the  reli¬ 
ability  and  maintainability  of  our  weapon 
systems  have  been  secondary  considerations 
in  the  acquisition  process.  It  is  time  to 
change  this  practice  ...." 

EMMA 

EMMA  (Expert  Missile  Maintenance  Aid) 
is  the  result  of  research  sponsored  by  the 
Air  Force  Armament  Laboratory,  Air  Force 
Systems  Command  at  Eglin  AFB,  Florida  and 
is  a  first  attempt  to  enhance  Ait  Force 
tactical  munition  maintenance  by  applying 
artificial  intelligence/expert  system 
technology  to  ATE.  The  objective  of  EMMA 
is  to  develop  an  automated  smart  munition 
test  system  that  augments  existing  ATE. 
The  ultimate  goal  of  EMMA  is  to  help  a 
novice  munition  technician  isolate  and 
diagnose  electronic,  electromechanical, 
and  mechanical  equipment  faults  to  the 
board/chassis  level  more  quickly  and  con¬ 
sistently  than  the  best  human  expert  using 
the  best  currently  available  ATE. 

EMMA  is  a  thirty  month  effort  split 
into  two  phases.  Phase  1  began  in  Septem¬ 
ber  1986  and  concluded  18  months  later  in 
July  1987.  This  phase  addressed  the 
field-level  maintenance  of  tactical  muni¬ 
tions  and  ultimately  resulted  in  two 
field-level  expert  system  prototypes. 
Phase  2  began  in  August  1987  and  is  sched¬ 
uled  to  conclude  in  April  1989,  28  months 
later.  Phase  2  focuses  on  depot-level 
maintenance  and  will  produce  two  depot- 
level  expert  system  prototypes.  Since 
depot-level  diagnostic  activities  are  more 
in-depth  and  detailed  than  the  field,  this 
phase  is  expected  to  be  more  difficult  and 
of  greater  complexity.  This  accounts  for 
the  greater  time  allotted  to  this  phase. 
The  prototypes  from  both  phases  are  tar¬ 
geted  towards  the  maintenance  technicians. 
Since  EMMA  is  constrained  by  schedule  and 
money,  the  number  of  tests  developed  under 
this  effort  is  limited,  yet  sufficient  to 
demonstrate  concept  feasibility  of  using 
expert  systems  for  munition  maintenance. 

EMMA  draws  on  many  different  types  of 
knowledge  and  information  to  perform  the 
diagnosis  of  the  faulty  munition.  The 
EMMA  knowledge  base  consists  of  mainte¬ 
nance  rules  or  Technical  Orders  (TOs), 


maintenance  technician  practices  (heuris¬ 
tics),  Unit  Under  Test  (UUT)  design,  ex¬ 
isting  test  equipment  capabilities,  fail¬ 
ure  rates,  and  test  costs.  This  knowledge 
is  gleaned  from  various  sources  including 
TOs,  schematics  of  the  UUT  and  the  test 
equipment,  knowledge  acquisition  inter¬ 
views  with  munition  maintenance  experts 
and  the  experts  that  designed  the  munition 
and  test  equipment.  Figure  1  depicts  how 
this  knowledge  is  brought  to  bear  on  the 
problem  of  diagnosing  the  faulty  munition. 
First,  the  symptoms  are  derived  from  the 
test  equipment  and  technician  observa¬ 
tions.  This  information  is  supplied  to 
the  expert  system  via  a  sophisticated, 
user-friendly  interface.  The  expert  sys¬ 
tem  then  employs  the  knowledge  stored  in 
the  knowledge  bases  and  derives  a  repair 
strategy  which  is  displayed  to  the  techni¬ 
cian  using  the  EMMA  computer. 


Figure  1.  EMMA  Expert  System 


EMMA  is  a  dual  contract  effort  per¬ 
formed  by  Raytheon  Company,  Missile  Sys¬ 
tems  Division  in  Bedford  Massachusetts, 
and  Rockwell  International  Corporation, 
Autonetics  Sensors  and  Aircraft  Systems 
Division  in  Anaheim,  California.  Both 
contractors  will  develop  a  phase  1  and 
phase  2  EMMA  prototype  resulting  in  a 
total  of  four  prototypes.  Both  contrac¬ 
tors  were  allowed  to  select  their  candi¬ 
date  vehicle  for  the  EMMA  program  within 
specified  limits.  Raytheon  selected  the 
AIM-7F  Sparrow  missile  as  their  candidate 
munition.  Rockwell  chose  the  GBU-15  modu¬ 
lar  glide  bomb. 

THE  A1H-7F  FIELD-LEVEL  EMMA  PROTOTYPE 

The  Raytheon  field-level  (phase  1) 
EMMA  prototype  was  designed  to  enhance  the 
field-level  maintenance  of  the  AIM-7F 
missile  by  augmenting  the  missile's  test 
set.  All  references  to  the  word  "EMMA"  in 
this  section  refer  to  the  Raytheon  AIM-7F 
version  of  EMMA.  The  field-level  test  set 
for  the  A1M-7F  is  the  AN/DSM-162  test  set. 
EMMA  is  hosted  on  a  Symbolics  3678  LISP 
machine  running  the  expert  system  shell 
ART  (Automated  Reasoning  Tool).  ART  pro- 


vides  a  production  language  that  is  pri¬ 
marily  rule-based.  Consequently,  EMMA  was 
developed  using  the  rule-based  approach. 
The  Symbolics  computer  is  connected  to  the 
AN/DSM-162  test  set  via  an  RS-232  cable. 
Figure  2  illustrates  the  major  components 
of  the  EMMA  system  and  how  they  are  inter¬ 
connected. 
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Figure  2.  AIM-7F  Field-level 
EMMA  Prototype 

The  RS-232  cable  allows  EMMA  to  oper¬ 
ate  in  three  modes  --  automatic,  semi¬ 
automatic,  and  manual.  The  distinguishing 
characteristics  of  these  modes  is  the 
level  of  automation  EMMA  is  allowed  during 
the  diagnostic  session.  The  automatic 
mode  uses  the  RS-232  interface  to  allow 
EMMA  to  direct  the  diagnostic  testing  and 
resequencing  of  tests.  EMMA  automatically 
accepts  data  from  the  test  set  via  the  RS- 
232  cable,  performs  the  fault  isolation, 
and  directs  the  test  set  to  perform  addi¬ 
tional  test,  if  required,  until  the  fault 
is  detected  or  all  tests  pass.  If  a  fault 
is  detected  during  automatic  operation, 
the  user  may  switch  to  semi-automatic  mode 
for  closer  control  over  the  testing  and 
the  ability  to  query  after  each  test  seg¬ 
ment. 

The  semi-automatic  mode  operates  sim¬ 
ilar  to  the  automatic  mode  with  one  excep¬ 
tion.  This  mode  stops  execution  of  EMMA 
at  the  completion  of  each  unique  test 
segment.  This  allows  the  technician  to 
query  EMMA  recommendations  using  the  ex¬ 
planation  capability.  Another  advantage 
of  the  automatic  modes  (semi  and  full)  is 


data  integrity.  Since  EMMA  passes  tne 
data  between  the  test  set  and  the  Symbol¬ 
ics  computer  via  the  RS-232  cable,  the 
data  are  more  likely  to  remain  valid  as 
opposed  to  transferring  data  via  a  techni¬ 
cian  who  could  inadvertently  introduce 
errors . 

The  last  mode  is  manual.  This  mode 
is  provided  in  case  an  RS-232  connection 
is  not  possible.  As  the  name  implies,  all 
activities  between  the  test  set  and  the 
Symbolics  computer  must  be  performed  man¬ 
ually  by  the  technician.  EMMA  will  direct 
the  technician  to  perform  the  appropriate 
actions  to  the  test  set  and  wait  for  the 
response.  The  technician  enters  the  re¬ 
sponses  from  the  test  set  into  EMMA. 

As  with  most  expert  systems,  EMMA  is 
able  to  explain  its  reasoning  process  to 
the  user  (technician).  EMMA  explains  its 
fault  detection  and  resequencing  logic. 
In  other  words,  EMMA  explains  a  detected 
fault  and  why  a  certain  test  is  being 
recommended.  Two  levels  of  explanation 
are  available  depending  upon  the  experi¬ 
ence  of  the  technician.  The  technician 
may  request  an  explanation  during  any 
phase  of  the  diagnostic  process.  This 
allows  the  technician  to  query  EMMA  during 
a  consultation  which  heightens  the  techni¬ 
cian's  understanding  of  what  EMMA  is  doing 
while  simultaneously  providing  the  techni¬ 
cian  with  a  valuable  training  aid. 

One  of  the  most  critical  aspects  of 
any  software  system  is  its  user-friendli¬ 
ness.  If  the  system  is  difficult  to  use 
and  the  user  does  not  use  it,  it  has 
failed.  EMMA  uses  windows  to  relay  infor¬ 
mation  to  the  technician  and  accepts  in¬ 
formation  via  menus.  Using  a  mouse,  the 
technician  is  able  to  enter  data  quickly 
and  accurately  without  having  to  learn 
cryptic  commands.  The  majority  of  the 
data  entered  into  EMMA  by  the  technician 
is  done  using  the  mouse;  however,  some 
keyboard  input  is  required.  Figure  3 
shows  the  screen  of  a  Symbolics  computer 
running  EMMA. 

THE  (30-15  FIBLD-LBVBL  BHHA  PROTOTYPE 

The  Rockwell  field-level  (phase  1) 
EMMA  prototype  was  designed  to  enhance  the 
field-level  maintenance  of  the  GBU-15 
glide  bomb  by  augmenting  the  field-level 
test  set  —  GJM-55.  All  references  to  the 
word  "EMMA"  in  this  section  refer  to  the 
Rockwell  GBU-15  version  of  EMMA  unless 
stated  otherwise.  EMMA  is  hosted  on  a  IBM 
PC/AT  compatible  computer  running  the 
expert  system  shell  M.l.  Although,  the 
M.l  language  is  primarily  rule-based,  EMMA 
was  developed  using  an  object  oriented 
approach.  The  rules  of  the  knowledge  base 
reference  objects  and  object  attributes. 
This  EMMA  did  not  support  the  capability 
for  an  automatic  mode  due  to  hardware 
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Figure  3.  AIM-7F  EMMA  Screen 


limitations  thereby  leaving  only  the  man¬ 
ual  mode  (i.e.,  no  connecting  cable). 
Figure  4  illustrates  the  major  components 
of  the  EMMA  system  and  how  they  are  inter¬ 
connected  . 

Uncertainty  is  addressed  in  this 
version  of  EMMA.  When  EMMA  asks  the  tech¬ 
nician  for  information,  the  technician  may 
enter  "unknown"  as  a  response.  EMMA  will 
accommodate  this  response  by  adapting  its 
reasoning  process  using  uncertainty.  Un¬ 
certainty  is  handled  using  a  MYCIN-like 
representation.  When  a  recommendation  is 
displayed  to  the  technician,  the  certainty 
of  the  recommendation  is  also  displayed  to 
indicate  the  belief  of  the  recommendation. 

The  GBU-15  EMMA  also  possesses  expla¬ 
nation  capabilities.  The  technician  may 
ask  EMMA  for  an  explanation  or  help  at  any 
time.  EMMA  will  respond  with  either  an 
explanation  of  the  reasoning  process  or 
information  that  will  guide  the  technician 
through  the  consultation.  The  explanation 
capability  can  handle  queries  regarding 
the  reason  a  certain  conclusion  was 
reached  or  why  EMMA  is  asking  for  informa¬ 
tion.  As  with  the  AIM-7F  EMMA,  the  GBU-15 
EMMA  has  two  levels  of  explanation  to 
accommodate  the  needs  of  different  techni¬ 
cians.  The  same  training  benefits  exist 
in  the  GBU-15  EMMA  as  the  AIM-7F  EMMA. 
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EMMA  exploits  the  use  of  pull-down 
menus  and  function  keys  on  the  computer  to 


Figure  4.  GBU-15  Fieid-level 
EMMA  Prototype 


make  it  as  user  friendly  as  possible.  The  Verification  of  ENHA 
majority  of  technician  interaction  with 

EMMA  is  performed  using  the  keyboard.  The  A  unique  aspect  of  the  verification 

technician  typically  responds  to  EMMA  of  the  EMMA  program  is  that  it  uses  DOD- 
questions  and  requests  with  short  answers  STD-2167,  the  Defense  System  Software 
thereby  reducing  the  probability  of  erro-  Development  standard,  to  develop  the  ex- 
neous  data  being  entered.  Figure  5  shows  pert  system  prototypes.  This  is  one  of 

the  screen  of  the  computer  running  EMMA.  the  first  attempts  to  apply  this  standard 

to  the  development  of  an  expert  system. 
EVALUATION  OF  THE  EMMA  PROTOTYPES  DOD-STD-2167  traditionally  employs  top- 

dov;n  development  of  large  software  syc- 
Meaningful  evaluation  of  expert  sys-  terns.  Expert  systems  on  the  other  hand 

terns  has  been  an  often  discussed  but  sel-  ate  developed  using  a  development  method- 

dom  achieved  topic  within  recent  years.  ology  that  is  less  rigidly  defined  which 

Mote  often  than  not  quantitative  metrics  typically  entails  an  iterative  approach  to 

are  simply  not  available  or  meaningful  as  software  development.  Additionally,  ex- 

an  evaluation  measure.  Since  an  expert  pert  systems  are  usually  created  by  a 

system  encapsulates  the  knowledge  of  a  relatively  small  team, 

given  expert  in  a  given  field,  the  effec¬ 
tive  evaluation  of  the  expert  system  may  The  EMMA  program  has  shown  that  ex- 

oe  difficult  at  best.  Validation  must  be  pert  systems  can  be  developed  using  DOD- 

used  to  justify  the  representation  levels  STD-2167  software  development  require- 

of  expert  systems  (O'Keefe  et  al.,  1987).  ments.  With  some  careful  tailoring  of 

some  of  the  documents,  this  standard  can 
Validation  is  typically  considered  a  be  effectively  used  to  document  the  pro¬ 
part  of  evaluation,  and  evaluation  is  gram  and  provide  the  program  manager  valu- 

concerned  with  determining  the  comprehen-  able  insight  into  the  contractor's  soft- 

sive  value  of  an  expert  system  (O'Keefe  et  ware  development,  testing,  and  evaluation 

al.,  1987).  Validation  should  not  be  efforts.  The  tailored  documents  were 

confused  with  verification.  "Validation  altered  to  accommodate  the  iterative  na- 

refers  to  building  the  right  system  (that  ture  of  expert  system  development, 

is,  substantiating  that  a  system  performs 

with  an  acceptable  level  of  accuracy).  Since  verification  must  determine 

whereas  verification  refers  to  building  whether  an  expert  system  correctly  imple- 

the  system  'right'  (that  is,  substanti-  ments  its  specifications,  testing  must 

ating  that  a  system  correctly  implements  occur  in  order  to  validate  this  require- 

its  specifications)"  (O'Keefe  et  al.,  ment.  Again,  DOD-STD-2167  proved  to  be 

1987).  Validation  and  verification  will  adequate  for  verification  testing  once 

be  addressed  in  this  paper  as  they  apply  extended.  Using  the  testing  procedures 

to  EMMA.  called  out  in  this  standard,  the  correct 


EMMA  CONFIG  NO:  33  TEST  NO:  0096  NOGO:  INV/M6  A1 

SRU  COMPONENT  CABLE  CERTAINTY 

INV/CONV  .  60% 

CONTROL  UNIT  (HARNESS)  UUT  INPUT  CABLE  053A20(BLU)  40% 

RECOMMENDATION: 

CHECK  THE  INPUT  CABLE  OF  THE  UNIT  UNDER  TEST.  IF  OK,  THEN 
R4R  INV/CONV 

JUSTIFICATION: 

A  REUTED  TEST  WITH  A  DIFFERENT  INPUT  CHANNEL  PASSED  AND 
THE  PARAMETERS  THAT  FAILED  WERE  APPROXIMATELY  ZERO:  THUS 
THE  INPUT  CABLES  OF  THE  UNIT  UNDER  TEST  COULD  BE  THE 
CAUSE.  A  MORE  LIKELY  FAILURE  IS  THE  INV/CONV  BECAUSE  THE 
+  28  VDC  ELECTRONICS  IS  COMPLEX  AND  COULD  CAUSE  ZERO 
FAILURES  AS  INDICATED 


Figure  5.  GBU-15  EMMA  Screen 


implementations  of  the  specifications  foe 
EMMA  were  verified.  Two  levels  of  testing 
occurred  to  accomplish  this  task.  First, 
informal  testing  took  place.  This  testing 
’erified  the  integrity  of  the  individual 
computer  software  units  before  the  units 
were  integrated  into  the  system  and  tested 
as  a  system.  Informal  testing  was  per¬ 
formed  by  the  knowledge  engineer.  Since 
expert  system  development  is  iterative  in 
nature,  informal  testing  essentially  oc¬ 
curs  throughout  development.  The  knowl¬ 
edge  engineer  and  the  expert  verify  the 
expert  system  and  identify  potential  cor¬ 
rections  and  enhancements.  Based  on  these 
recommendations,  the  knowledge  engineer 
was  able  to  incorporate  the  recommenda¬ 
tions.  Second,  formal  testing  occurred. 
An  independent  team  performed  the  formal 
testing  by  exercising  EMMA  using  test 
plans  and  test  descriptions  generated 
using  DOD-STD-2167. 

Validation  of  EMMA 

The  validation  of  EMMA  will  be  ad¬ 
dressed  in  this  paper  in  two  areas. 
First,  the  performance  validation  of  EMMA 
will  be  discussed  (i.e.,  how  well  EMMA 
performed).  Second,  the  human  factors 
aspect  of  validation  will  be  addressed. 
Both  areas  are  extremely  important  to  the 
success  of  an  expert  system.  The  follow¬ 
ing  paragraphs  will  present  the  validation 
of  the  two  EMMA  prototypes.  The  valida¬ 
tion  methodology  will  be  discussed  fol¬ 
lowed  by  the  results  of  the  validation. 

As  with  most  expert  systems,  the 
ultimate  measure  of  success  is  determined 
when  the  system  is  used  by  the  end  users 
(in  this  case  the  field-level  munition 
technician).  This  is  the  approach  taken 
with  the  EMMA  program.  Both  contractors 
took  their  respective  prototypes  to  Air 
Force  bases  in  which  their  selected  muni¬ 
tion  is  used  and  a  field-level  maintenance 
capability  exists.  This  allowed  the  pro¬ 
totypes  to  be  evaluated  in  an  actual  field 
test  environment. 

After  considering  several  alterna¬ 
tives,  both  contractors  decided  to  use  a 
toggle  switch  box  to  insert  faults  into  a 
known  good  missile.  This  approach  was 
necessary  since  it  was  feared  that  the 
maintenance  squadrons  in  the  field  might 
not  have  a  sufficient  number  of  faulty 
munitions  during  the  evaluation  period. 
These  faults  were  induced  by  the  user  by 
simply  toggling  one  of  the  switches  which 
in  turn  would  disturb  one  or  more  signals 
within  the  munition.  The  faults  were 
defined  by  the  domain  expert  such  that  the 
faults  would  adequately  exercise  the  var¬ 
ious  characteristics  of  the  EMMA  proto¬ 
types  which  included  the  resequencing 
logic,  the  explanation  capability,  and  the 
fault  isolation  logic.  To  verify  the 
expert  was  not  trying  to  select  only  the 


faults  EMMA  could  handle  best,  the  mainte¬ 
nance  experts  agreed  the  selected  faults 
were  representative  of  faults  they  common¬ 
ly  experience. 

The  AIN-7F  EMMA  Evaluation.  Raytheon 
took  their  field  level  AIM-7F  prototype  to 
the  325th  Equipment  Maintenance  Squadron 
(EMS)  at  Tyndall  AFB,  Florida  for  an  eval¬ 
uation  period  that  began  on  8  June  1987 
and  concluded  on  12  June  1987. 

The  argument  could  be  made  that  EMMA 
should  accurately  diagnose  all  the  induced 
faults  since  the  expert  system  and  the 
faults  were  derived  from  the  same  source 
—  the  domain  expert.  In  order  to  demon¬ 
strate  the  robustness  of  EMMA,  an  addi¬ 
tional  evaluation  methodology  was  used. 
Two  faulted  missiles  were  saved  by  the  EMS 
prior  to  the  evaluation.  These  missiles 
had  previously  failed  testing  using  the 
AN/DSM-162  test  set.  However,  the  fault 
data  for  these  missiles  were  not  released 
by  the  EMS  personnel  until  after  the  EMMA 
evaluation.  A  third  missile  became  avail¬ 
able  during  the  EMMA  evaluation  by  failing 
a  flight  line  test  during  prelaunch  tun¬ 
ing.  This  missile  was  an  excellent  exer¬ 
cise  for  the  EMMA  prototype  since  it  was 
not  previously  tested  by  the  AN/DSM-162 
test  set.  Its  fault  was  unknown  to  every¬ 
one  present  at  the  evaluation.  All  three 
missiles  (sometimes  referred  to  as  "mys¬ 
tery  missiles"  due  to  their  unknown  past) 
contained  faults  unknown  to  EMMA  or  the 
domain  expert  thereby  exercising  EMMA  in 
an  unpredictable  manner. 

Four  munition  maintenance  technicians 
from  the  EMS  at  Tyndall  were  used  for  the 
evaluation.  Two  technicians  were  classi¬ 
fied  as  novice  with  little  experience  with 
the  AN/DSM-162  test  set  and  its  associated 
operating  procedures.  The  other  two  tech¬ 
nicians  were  classified  as  experts  with  a 
substantial  background  in  using  the 
AN/DSM-162  test  set.  Two  teams  of  two 
technicians  were  created  consisting  of  one 
expert  and  one  novice.  One  team  (hereaf¬ 
ter  referred  to  as  the  EMMA  team)  received 
extensive  training  on  the  operation  of 
EMMA.  The  other  team  (hereafter  referred 
to  as  the  non-EMMA  team)  was  not  trained 
on  the  EMMA  system  and  served  as  a  base¬ 
line  for  the  evaluation. 

Twelve  faults  were  inserted  into  a 
known  good  missile  using  the  toggle  switch 
box.  The  faults  were  diagnosed  by  the 
EMMA  team  using  the  EMMA  system  and  the 
non-EMMA  team  using  just  the  AN/DSM-162 
test  set  and  the  applicable  TO.  Perfor¬ 
mance  of  the  two  teams  was  based  on  the 
level  of  expertise  of  the  operator,  dura¬ 
tion  of  test,  and  the  ability  to  diagnose 
the  fault  accurately. 


The  results  of  this  evaluation  exer¬ 
cise  were  very  promising.  There  were 
three  significant  results  derived  from  the 
evaluation.  First,  the  EMMA  system  oper¬ 
ated  by  the  EMMA  team  was  able  to  consis¬ 
tently  diagnose  the  fault  quicker  than  the 
non-EMMA  team  using  just  the  AM/DSM-162 
and  the  TO  regardless  of  the  experience 
level  of  the  EMMA  operator.  A  time  sav¬ 
ings  of  20%  was  seen  with  the  novice  using 
EMMA  over  the  expert  using  the  AN/DSM-162. 
Second,  novice  technicians  using  the  EMMA 
system  significantly  outperformed  (better 
fault  diagnoses)  novice  technicians  using 
just  the  AN/DSM-162  and  performed  33% 
better  than  expert  technicians  using  just 
the  AN/DSM-162.  Finally,  EMMA'S  explana¬ 
tion  capabilities  significantly  enhanced 
the  abilities  of  the  EMMA  team  to  deter¬ 
mine  the  reason  behind  each  fault. 

Once  EMMA'S  abilities  were  exercised 
using  the  induced  faults,  EMMA  was  pitted 
against  the  mystery  missiles  again  with 
excellent  results.  The  EMMA  team  using 
EMMA  correctly  isolated  the  faults  in  all 
three  missiles.  Only  after  EMMA  diagnosed 
the  faults  was  the  previous  testing  data 
on  the  missiles  released.  EMMA'S  diagno¬ 
sis  was  consistent  with  this  data. 

User  acceptance  of  EMMA  was  outstand¬ 
ing.  in  fact,  the  technicians  accepted 
EMMA'S  diagnosis  of  the  missile  from  the 
flight  line  and  said  they  would  have,  if 
allowed,  sent  the  missile  to  the  depot 
with  no  further  testing  using  the  AN/DSM- 
162  test  set.  This  exemplifies  EMMA'S 
acceptance  by  the  EMS  maintenance  person¬ 
nel  at  Tyndall  AFB,  The  technicians  found 
the  system  to  be  very  user-friendly.  The 
mouse  and  the  use  of  menus  made  the  system 
easier  to  use  thaii  the  bulky  and  cumber¬ 
some  TOs.  Also,  the  explanation  capabili¬ 
ty  proved  to  be  an  effective  training 
mechanism. 

Ttie  (SU-IS  EMMA  Evaluation.  Rockwell 
evaluated  their  GBU-15  EMMA  at  the  4th 
Equipment  Maintenance  Squadron  located  at 
Seymour  Johnson  AFB,  North  Carolina  during 
the  period  of  22  June  through  29  June 
1987. 

Four  maintenance  technicians  were 
used  in  the  evaluation  of  the  EMMA  proto¬ 
type.  Two  technicians  were  considered 
experts  with  several  years  of  experience 
with  the  GBU-15  test  environment.  The 
remaining  two  technicians  were  considered 
novices  with  less  than  6  months  of  expe¬ 
rience.  Another  important  distinction 
between  the  expert  and  novice  technicians 
is  the  fact  that  the  expert  technicians 
owned  personal  computers  and  therefore 
were  familiar  with  how  computers  operate 
whereas  the  novice  technicians  did  not  own 
computers  and  had  never  used  a  computer 
before  the  EMMA  evaluation.  All  four 
technicians  were  trained  on  how  to  use  the 


EMMA  system.  After  this  brief  training, 
the  technicians  felt  very  comfortable 
using  the  system. 

Twenty-two  simulated  faults  were  in¬ 
duced  into  the  known  good  munition  with 
the  intent  of  evaluating  EMMA'S  capabili¬ 
ties  to  handle  the  following  five  areas: 
resolution  of  ambiguities  between  major 
shop  replaceable  units  (SRU) ,  referencing 
lower  configuration  testing  to  facilitate 
further  component  resolution,  distin¬ 
guishing  between  a  cable  failure  and  a 
circuit  card  assembly  (CCA)  gain  failure, 
resolution  of  ambiguities  between  CCA's, 
and  recognizing  operator  errors  or  test 
set  problems.  Six  of  the  twenty-two  in¬ 
duced  faults  were  in  the  all-up-round 
(AUR)  configuration  (i.e.,  the  test  was 
performed  while  the  GBU-IS  munition  was 
completely  intact).  The  remaining  sixteen 
faults  were  in  the  control  module  stand 
alone  configuration.  EMMA  was  able  to 
handle  these  five  areas  by  analyzing  addi¬ 
tional  test  parameters  as  well  as  insti¬ 
tuting  and  analyzing  tests  related  to  the 
failed  test. 

The  diagnostic  results  of  the  induced 
faults  showed  substantial  time  savings  in 
fault  isolation  and  increased  diagnostic 
capabilities.  While  the  munition  was  in 
the  control  module  stand  alone  configura¬ 
tion,  a  time  savings  of  40%  was  seen  over 
conventional  testing  with  the  GJM-55. 
When  the  munition  was  in  the  AUR  configu¬ 
ration,  EMMA  was  able  to  provide  up  to  74% 
time  savings.  This  is  due  to  EMMA's  capa¬ 
bility  to  resolve  failures  while  the  muni¬ 
tion  is  in  the  AUR  configuration  thereby 
saving  the  technician  from  having  to  per¬ 
forming  testing  in  stand  alone  configura¬ 
tion. 

The  GJM-55  test  set,  in  some  situa¬ 
tions,  will  recommend  more  than  one  sus¬ 
pected  failure.  This  group  of  failures  is 
called  an  ambiguity  group  since  the  test 
set  cannot  resolve  any  further  than  this 
group.  This  is  another  of  EMMA'S  capabil¬ 
ities  that  demonstrated  promising  perfor¬ 
mance  as  seen  by  the  results  that  follow. 
EMMA  also  considered  the  possibility  of  a 
cable  harness  failure  or  the  test  sec  is 
failing.  Based  on  these  capabilities  EMMA 
was  able  to  significantly  improve  fault 
isolation.  The  results  of  the  twenty-two 
simulated  faults  demonstrate  this  improve¬ 
ment.  EMMA  added  a  wiring  harness  check 
to  50%  of  all  tests.  EMMA  deleted  a  CCA 
from  an  ambiguity  group  40%  of  the  time 
thereby  reducing  the  number  of  CCA  to  be 
considered  during  testing.  EMMA  added  a 
CCA  to  an  ambiguity  group  30%  of  the  time 
to  insure  all  potential  CCA's  ace  con¬ 
sidered  during  the  testing.  This  indi¬ 
cates  that  the  test  set  sometimes  did  not 
consider  all  potential  CCA's.  Finally, 
EMMA  exchanged  one  suspect  CCA  in  an  ambi¬ 
guity  group  for  another  CCA  10%  of  the 


time.  The  ability  to  manipulate  the  ambi¬ 
guity  group  to  benefit  fault  isolation  was 
demonstrated  by  EMMA  and  proved  to  be  an 
effective  fault  isolation  technique. 
Tnese  results  directly  support  the  time 
savings  previously  mentioned. 

The  GBU-15  EMMA  prototype  also  re¬ 
ceived  accolades  for  its  user  friend¬ 
liness.  The  technicians  used  EMMA  with 
comfort  and  found  several  items  to  be 
particularly  laudable.  Among  these  items 
was  the  understandabi 1 ity  of  EMMA.  The 
explanation  capability  provided  easy  to 
understand  responses.  Another  aspect  they 
found  beneficial  was  the  addition  of  the 
internal  wiring  harness  check  as  one  of 
the  reasons  for  a  fault  since  this  check 
IS  relatively  "inexpensive"  to  perform  and 
can  prevent  unnecessary  and  potentially 
costly  future  testing.  The  training  po¬ 
tential  of  EMMA  was  also  mentioned  as  one 
of  its  magor  assets  with  the  shortage  of 
skilled  technicians  in  the  munition  main¬ 
tenance  field. 

EHHA  PHASE  2 

Both  contractors  are  currently  in 
phase  2  of  the  EMMA  program.  As  previous, 
ly  mentioned,  phase  2  focuses  on  the  main¬ 
tenance  of  tactical  munitions  at  the  depot 
level.  Mote  specifically,  Raytheon  is 
focusing  on  the  depot-level  maintenance  of 
the  AIM-7P.  Rockwell  is  using  the  GBW-Z5 
as  its  depot-level  maintenance  munition. 
Phase  2  is  a  natural  extension  of  phase  1 
since  field-level  faults  are  sent  to  the 
depot  for  repair.  The  prototypes  devel¬ 
oped  during  the  phase  2  effort  will  be 
more  detailed  extensions  of  the  phase  1 
prototypes  with  one  exception;  the  phase  2 
prototypes  will  augment  the  depot-level 
test  sets.  The  depot  test  set  for  the 
AIM-7F  is  the  AN/DPM-22  test  set.  The 
GBO-15  depot-level  test  set  is  CATS  (Cal¬ 
culator  Automatic  Test  Station). 

The  depot-level  prototypes  will  be 
implemented  on  the  same  computer  hardware 
using  the  same  expert  system  shells  as  the 
field-level  prototypes.  However,  one  dif¬ 
ference  between  the  field  and  depot  proto¬ 
types  for  both  contractors  is  the  inter¬ 
face  between  the  test  set  and  the  EMMA 
computer.  The  Raytheon  interface  will 
only  support  one-way  communication  from 
the  test  set  to  the  EMMA  computer  due  to 
test  set  limitations.  This  is  different 
than  the  two-way  communication  of  the 
field  prototype.  Rockwell  is  using  a  two- 
way  communication  interface  between  the 
test  set  and  the  EMMA  computer  whereas  the 
field  prototype  interface  was  manual. 
Both  prototypes  will  again  incorporate  an 
explanation  capability  for  the  techni¬ 
cians. 


The  evaluation  of  the  depot  proto¬ 
types  will  follow  the  same  methodology 
used  in  phase  1.  Each  prototype  will  be 
evaluated  at  the  actual  depot  location  by 
actual  depot  technicians.  Once  again, 
both  evaluations  are  scheduled  to  last 
five  days  and  are  scheduled  to  occur  in 
February  1989.  The  AIM-7F  prototype  will 
be  evaluated  at  the  Naval  Aviation  Depot 
(NAVAVNOEP)  in  Alameda,  California.  The 
GBU-IS  prototype  will  be  evaluated  at 
Rockwell's  Missile  Systems  Division  in 
Atlanta,  Georgia  since  an  organic  depot 
capability  currently  does  not  exist. 

FOLLOH-ON  PROGHAHS  TO  EMMA  —  EMMA  2 

A  follow-on  program  will  be  initiated 
in  early  1990  --  EMMA  2.  The  primary 

thrust  of  EMMA  2  is  to  develop  an  expert 
system  that  is  capable  of  diagnosing  a 
family  of  tactical  munitions  at  the  depot 
level.  The  current  EMMA  is  limited  to  one 
munition  per  prototype.  EMMA  2  would 
attempt  to  expand  the  current  prototype 
capabilities  to  include  multiple  munitions 
from  the  same  family  (e.g.,  AIM  family, 
GBU  family,  surface-to-air  family,  etc.). 
EMMA  2  would  draw  on  the  best  features  of 
all  prototypes  developed  in  the  two  phases 
of  EMMA  to  derive  a  robust  system. 

OTHER  ISSOES/OBSERVATIOHS 

The  following  paragraphs  present  oth¬ 
er  areas  of  interest  to  the  EMMA  program. 

Studies 

As  part  of  the  phase  1  effort,  five 
studies  were  conducted  to  address  areas  of 
concern  that  could  be  incorporated  into 
the  depot-level  prototypes  and  potentially 
in  future  maintenance  expert  systems. 
These  studies  included  the  reuse  of  muni¬ 
tion  test  programs,  the  use  of  the  Ada 
language  for  expert  system  development  and 
ATE  test  programs,  the  applicability  of 
the  Modular  Automatic  Test  Equipment 
(MATE)  standard  to  EMMA,  the  applicability 
of  the  Warner  Robins  Reliability  Asset 
Monitor  (RAM)  database  to  EMMA  and  future 
maintenance  expert  systems,  and  the  secu¬ 
rity  issues  of  expert  systems.  A  complete 
discussion  of  the  results  of  these  studies 
is  beyond  the  scope  of  this  paper.  There¬ 
fore,  the  interested  reader  is  referred  to 
the  two  final  reports  of  phase  1  (Elerin 
et  al.f  1987;  Davis,  1987).  These  final 
reports  are  split  into  two  volumes;  the 
second  volume  contains  a  complete  discus¬ 
sion  of  the  results  of  these  studies. 

Current  Maintenance  Philosophy 

The  current  munition  maintenance  phi¬ 
losophy  of  the  Tactical  Air  Command  (TAC) 
for  field  maintenance  is  that  of  fault 
detection  (go/nogo  testing).  If  a  fault 
does  occur  in  the  field,  the  suspected 


faulty  secti.n  of  the  missile  is  sent  to 
the  depot  repair.  One  of  the  driving 
factors  ot  this  philosophy  is  the  shortage 
of  skilled  maintenance  technicians  in 
field-level  maintenance. 

Since  training  these  technicians  is 
costly,  TAG  decided  to  eliminate  an  Air 
Force  Specialty  Code  (AFSC)  for  munition 
maintenance.  The  deleted  AFSC,  316X1L, 
was  an  electronics  munition  maintenance 
specialist.  With  this  specialist  no  lon¬ 
ger  available,  munition,  not  electronic 
munition,  specialist  are  diagnosing  to¬ 
day's  munitions.  This  tends  to  create 
problems.  The  munition  specialists  are 
typically  not  adequately  trained  to  diag¬ 
nose  the  electronically-sophisticated  mu¬ 
nitions  of  today. 

EMMA  is  capable  of  providing  the 
necessary  training  of  munition  maintenance 
technicians.  Using  the  explanation  capa¬ 
bilities  of  EMMA,  a  technician  can  quickly 
become  skilled  at  diagnosing  the  munition. 
Since  EMMA'S  knowledge  is  gleaned  from 
diagnostic  and  design  experts,  the  novice 
munition  technician  using  EMMA  will  effec¬ 
tively  be  performing  as  if  he  has  an  ex¬ 
pert  maintenance  technician,  the  designer 
of  the  test  set,  the  munition  designer, 
and  an  instructor  looking  over  his  shoul¬ 
der  during  the  diagnosis.  Another  aspect 
of  EMMA  that  should  reduce  overall  mainte¬ 
nance  costs  is  its  ability  to  diagnose  to 
a  greater  component  level  than  existing 
test  sets  used  by  today's  technicians. 
This  should  significantly  reduce  the  costs 
associated  with  shipping  faulty  munitions 
to  the  depot,  since  more  faults  can  be 
isolated  at  the  field. 

The  munition  depots  are  currently 
responsible  for  the  majority  of  munition 
repairs.  The  field  has  very  limited  re¬ 
pair  capabilities.  This  results  in  in¬ 
creased  costs  for  overall  maintenance  of  a 
munition  system.  The  obvious  cost  is 
transportation  of  the  munition  between  the 
field  and  the  depot.  However,  the  less 
tangible  and  potentially  more  significant 
cost  is  that  of  having  the  munition  out  of 
the  inventory.  This  effectively  reduces 
the  number  of  missiles  available  for  exer¬ 
cises  or  conflict. 

Future  Hainteoance  Systaas 

EMMA  prototypes  have  proved  the  fea¬ 
sibility  of  applying  AI  to  munition  main¬ 
tenance.  In  future  weapon  systems,  main¬ 
tenance  expert  systems  should  evolve  with 
the  weapon  system  instead  of  after  the 
fact.  This  would  allow  the  expert  system 
to  capture  knowledge  about  the  weapon  as 
it  is  developed.  Furthermore,  the  expert 
system  should  be  incorporated  directly 
into  the  ATE  instead  of  augmenting  the  ATE 
with  a  separate  computer  system.  This 
should  make  weapon  systems  of  the  future 


more  supportable  by  considering  the  main¬ 
tenance  aspect  early  in  the  weapon  life 
cycle. 

SUMHART 

Tactical  munition  maintenance  of  to¬ 
day  has  problems.  EMMA  is  an  attempt  to 
relieve  some  of  these  problems  by  applying 
artificial  intelligence/expert  system 
technology.  The  results  of  EMMA  indicate 
that  this  approach  to  munition  maintenance 
has  significant  potential  for  future  tac¬ 
tical  maintenance  systems. 

Corporate  knowledge  retention  is  one 
of  the  premium  benefits  of  EMMA.  Since 
EMMA  is  updated  easily  and  it  never  "for¬ 
gets"  knowledge,  EMMA  is  an  excellent  tool 
for  storing  corporate  knowledge  as  techni¬ 
cians  come  and  go.  Also,  EMMA  provides 
consistent,  high  quality  diagnosis  since 
it  never  has  a  "bad"  day  as  contrasted 
with  technicians.  Rapid  fault  isolation 
and  efficient  manpower  utilization  are  two 
more  benefits  of  using  EMMA.  These  bene¬ 
fits  provided  by  EMMA  will  result  in  sud- 
stantial  mission  payoffs.  Weapon  system 
downtime  will  be  decreased  as  well  as 
personnel  requirements  and  training  time. 
However,  the  most  significant  payoff  is 
the  increase  in  the  reliability  of  muni¬ 
tion  maintenance  procedures. 
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